Not all computational problems utilize the types of parallel applications traditionally designed to run on high-performance computing (HPC) systems. Today, many workloads running on these systems often require a modest amount of computing resources for any given job or task. For specific research workloads, however, a more important consideration is actually how much aggregate compute power can be consistently and reliably leveraged against a problem over time. These high-throughput computing (HTC) workloads aim to solve larger problems over extended periods by completing numerous smaller computational subtasks. For example, these often involve significant parameter sweeps over simulation input parameters or regular processing and analysis of data collected from specialized instruments. In some cases, these problems are also composed of numerous district computational subtasks linked together in highly structured, complex workflows, which can become a challenge in and of themselves to design and manage effectively. If your research problem can leverage a high-throughput or many-task computing (MTC) model, then it is vital to learn how to build and run these types of workflows safely and effectively on HPC systems.
In this third part of our series on Batch Computing, we introduce you to high-throughput and many-task computing using the Slurm Workload Manager. In particular, you will learn how to use Slurm job arrays and job dependencies, which can be used to create these more structured computational workflows. We will also highlight some of the problems you’ll likely come across when you start running HTC and/or MTC workloads on HPC systems. This will include a discussion on job bundling strategies — what they are and when to use them. Additional topics about high-throughput and many-task computing workflows will be covered as time permits
High Throughput and Many Task Computing (COMPLECS)
Remote event
Instructor
Marty Kandes, Ph.D.
Computational & Data Science Research Specialist, SDSC
Marty Kandes is a Senior Computational and Data Science Research Specialist at the San Diego Supercomputer Center (SDSC). As part of the High-Performance Computing (HPC) User Services Group within the Data-Enabled Scientific Computing Division, he provides technical user support and services to the national research community leveraging the Advanced Cyberinfrasurcture (CI) and HPC resources designed, built and operated by SDSC on behalf of the U.S. National Science Foundation (NSF). Marty is also a member of the National Artificial Intelligence (AI) Research Institute for Intelligent CI with Computational Learning in the Environment (ICICLE). His current research interests include problems in distributed AI inference over wireless networks, data privacy in natural language processing, and secure interactive computing. He also contributes to many of the education, outreach, and training initiatives at SDSC, including serving as a Co-PI for the COMPrehensive Learning for end-users to Effectively utilize CyberinfraStructure (COMPLECS) CyberTraining program and as mentor for the Research Experience for High School Students (REHS) program. Marty received his Ph.D. in Computational Science from the Computational Science Research Center (CSRC) at San Diego State University (SDSU), where he studied quantum systems in rotating frames of reference through the use of numerical simulations. He also holds an M.S. in Physics from SDSU and dual B.S. degrees in Applied Mathematics and Physics from the University of Michigan, Ann Arbor.